Learning a decision tree from data is a difficult optimization problem. The most widespread algorithm in practice, dating to the 1980s, is based on a greedy growth of the tree structure by recursively splitting nodes, and possibly pruning back the final tree. The parameters (decision function) of an internal node are approximately estimated by minimizing an impurity measure. We give an algorithm that, given an input tree (its structure and the parameter values at its nodes), produces a new tree with the same or smaller structure but new parameter values that provably lower or leave unchanged the misclassification error. This can be applied to both axis-aligned and oblique trees and our experiments show it consistently outperforms various other algorithms while being highly scalable to large datasets and trees. Further, the same algorithm can handle a sparsity penalty, so it can learn sparse oblique trees, having a structure that is a subset of the original tree and few nonzero parameters. This combines the best of axis-aligned and oblique trees: flexibility to model correlated data, low generalization error, fast inference and interpretable nodes that involve only a few features in their decision.

Thismakesinferenceveryfast,andmaynot even need to use all input featuresto make a prediction (with axis-aligned trees).

's leaf nodes to form Given the definition of our attention in Eq. 9 in the main text, the highest polynomial order is Before providing the proof of Theorem 4, we establish Lemma 1 as its foundation. We follow the principle of Y an et al's work [ Figure 1, we consider two kinds of value functions, i.e., In P AM-Net, we set the number of levels to 2. A grid search is performed over different configurations We conduct grid searches on the dropout rate over {0, 0.1, 0.2} and the initial

Learning feature interactions can be the key for multivariate predictive modeling. ReLU-activated neural networks create piecewise linear prediction models.

Neural Information Processing Systems http://nips.cc/